Braille generator and converter

ABSTRACT

Techniques described herein relate to generating braille output and/or visual display output based on received mathematical expression input. Data corresponding to one or more mathematical expressions may be received via expression input devices or visual display devices, and may be converted to braille output characters for display on refreshable braille devices. Additionally, mathematical expression input data may be received via refreshable braille display devices and converted to output characters for display on visual display devices. In some embodiments, mathematical expression input data may be converted first to content markup, and then converted from the content markup to presentation markup and/or braille output characters. Further mathematical expression input data, such as updates to previously displayed expressions, may be received from an initial input device, visual display device, or refreshable braille device, and the updated expression data may be converted to visual output characters and/or braille output characters.

BACKGROUND

A number of computing devices and technologies are available forvisually impaired users who are unable to read monitors and other visualdisplay screens. For example, refreshable braille display devices mayinclude a number of electromechanical braille cells having output pinsthat may be raised or lowered to form braille characters. Such devicesalso may include braille keyboards to allow for braille readers to inputdata. Additional devices used by visually impaired users may includedevices having microphones and voice recognition and controlfunctionality for receiving input, as well as device speakers and speechsynthesizers for generating audio rather than visual output.

BRIEF SUMMARY

Various techniques are described herein for generating visual displayoutput and/or braille output based on received mathematical expressioninput. In certain embodiments, data corresponding to one or moremathematical expressions may be received via expression input devices orvisual display devices, and may be converted to braille outputcharacters for display on one or more refreshable braille devices.Additionally, mathematical expression input data may be received via oneor more refreshable braille display devices and converted to outputcharacters for display on visual display devices. In some embodiments,mathematical expression input data may be converted first to contentmarkup, and then converted from the content markup to presentationmarkup and/or braille output characters. Further mathematical expressioninput data, such as updates to previously displayed expressions, may bereceived from an initial input device, visual display device, orrefreshable braille device, and the updated expression data may beconverted to visual output characters and/or braille output characters.

In certain techniques described herein, at least four separate dataconverters and/or conversion processes may be supported to perform fourseparate expression conversions, including a mathematical expressioninput to content markup converter and/or conversion process, a braillecharacter input to content markup converter and/or conversion process, acontent markup to braille output character converter and/or conversionprocess, and/or a content markup to presentation markup converter and/orconversion process. In some embodiments, each mathematical expressionconverter and/or conversion process may use separate sets of conversionrules, including character mapping rules as well as various specialencoding and decoding rules for mathematical expressions. Additionally,in certain embodiments, some or all of the converters and/or conversionprocesses may be initiated in real-time or near real-time in response toreceiving a single-character input or revision to a mathematicalexpression from a visual display device or a refreshable brailleterminal. The implementation of these embodiments and other embodimentsdescribed herein may provide support for multi-user viewing, editing,and other collaborative work involving mathematical expression data,where such multi-user collaboration may occur simultaneously and withoutexpression ambiguity, for combinations of sighted users interacting viavisual display devices and visually impaired users interacting viarefreshable braille display devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing illustrating an example of a contentdistribution network.

FIG. 2 is a block diagram illustrating a computer server and computingenvironment within a content distribution network.

FIG. 3 is a block diagram illustrating an embodiment of one or more datastore servers within a content distribution network.

FIG. 4 is a block diagram illustrating an embodiment of one or morecontent management servers within a content distribution network.

FIG. 5 is a block diagram illustrating the physical and logicalcomponents of a special-purpose computer device within a contentdistribution network.

FIG. 6 is a block diagram illustrating an example system for convertingand generating braille mathematical expression output and visualmathematical expression output, according to one or more embodiments ofthe disclosure.

FIG. 7 is a flow diagram illustrating an example process of convertingmathematical expression data received from an input device to brailleoutput and/or visual display output, according to one or moreembodiments of the disclosure.

FIG. 8 is a flow diagram illustrating an example process of convertingmathematical expression input data received from a braille device tobraille output and/or visual display output, according to one or moreembodiments of the disclosure.

FIG. 9 is a flow diagram illustrating an example process of convertingmathematical expression input data received from a visual display deviceto braille output and/or visual display output, according to one or moreembodiments of the disclosure.

FIGS. 10A and 10B show samples of content markup and presentation markuprepresenting an example mathematical expression, according to one ormore embodiments of the disclosure.

FIGS. 11A and 11B show samples of content markup and presentation markuprepresenting another example mathematical expression, according to oneor more embodiments of the disclosure.

FIGS. 12, 13, and 14A-14C show various examples of visually displayedmathematical expressions along with corresponding braille output,according to one or more embodiments of the disclosure.

In the appended figures, similar components and/or features may have thesame reference label. Further, various compo of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

The ensuing description provides illustrative embodiment(s) only and isnot intended to limit the scope, applicability or configuration of thedisclosure. Rather, the ensuing description of the illustrativeembodiment(s) will provide those skilled in the art with an enablingdescription for implementing a preferred exemplary embodiment. It isunderstood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

Various techniques (e.g., systems, methods, computer-program productstangibly embodied in a non-transitory computer-readable storage medium,etc.) are described herein for generating visual display output and/orbraille output based on received mathematical expression input. Incertain examples described herein, data corresponding to one or moremathematical expressions may be received via expression input devices orvisual display devices, and may be converted to braille outputcharacters for display on a refreshable braille device. Additionally,mathematical expression input data may be received via one or morerefreshable braille display devices and converted to output charactersfor display on visual display devices. In some embodiments, mathematicalexpression input data may be converted first to content markup, and thenconverted from the content markup to presentation markup and/or brailleoutput characters. Further mathematical expression input data, such asupdates to previously displayed expressions, may be received from aninitial input device, visual display device, or refreshable brailledevice, and the updated expression data may be converted to visualoutput characters and/or braille output characters.

In accordance with certain techniques described herein, at least fourseparate data converters and/or conversion processes may be supported toperform four separate expression conversions, including a mathematicalexpression input to content markup converter and/or conversion process,a braille character input to content markup converter and/or conversionprocess, a content markup to braille output character converter and/orconversion process, and/or a content markup to presentation markupconverter and/or conversion process. In some embodiments, eachmathematical expression converter and/or conversion process may useseparate sets of conversion rules, including various special encodingand decoding rules for mathematical expressions that do not correspondto character mapping rules. Additionally, in certain embodiments, someor all of the converters and/or conversion processes may be initiated inreal-time or near real-time in response to receiving a single-characterinput or revision to a mathematical expression from a visual displaydevice or a refreshable braille terminal. The implementation of theseembodiments and other embodiments described herein may provide supportfor multi-user viewing, editing, and other collaborative work involvingmathematical expression data, where such multi-user collaboration mayoccur simultaneously and without expression ambiguity, for combinationsof sighted users interacting via visual display devices and visuallyimpaired users interacting via refreshable braille display devices.

With reference now to FIG. 1, a block diagram is shown illustratingvarious components of a content distribution network (CDN) 100 whichimplements and supports certain embodiments and features describedherein. Content distribution network 100 may include one or more contentmanagement servers 102. As discussed below in more detail, contentmanagement servers 102 may be any desired type of server including, forexample, a rack server, a tower server, a miniature server, a bladeserver, a mini rack server, a mobile server, an ultra-dense server, asuper server, or the like, and may include various hardware components,for example, a motherboard, a processing units, memory systems, harddrives, network interfaces, power supplies, etc. Content managementserver 102 may include one or more server farms, clusters, or any otherappropriate arrangement and/or combination or computer servers. Contentmanagement server 102 may act according to stored instructions locatedin a memory subsystem of the server 102, and may run an operatingsystem, including any commercially available server operating systemand/or any other operating systems discussed herein.

The content distribution network 100 may include one or more data storeservers 104, such as database servers and file-based storage systems.Data stores 104 may comprise stored data relevant to the functions ofthe content distribution network 100. Illustrative examples of datastores 104 that may be maintained in certain embodiments of the contentdistribution network 100 are described below in reference to FIG. 3. Insome embodiments, multiple data stores may reside on a single server104, either using the same storage components of server 104 or usingdifferent physical storage components to assure data security andintegrity between data stores. In other embodiments, each data store mayhave a separate dedicated data store server 104.

Content distribution network 100 also may include one or more userdevices 106 and/or supervisor devices 110. User devices 106 andsupervisor devices 110 may display content received via the contentdistribution network 100, and may support various types of userinteractions with the content. User devices 106 and supervisor devices110 may include mobile devices such as smartphones, tablet computers,personal digital assistants, and wearable computing devices. Such mobiledevices may run a variety of mobile operating systems, and may beenabled for Internet, e-mail, short message service (SMS), Bluetooth®,mobile radio-frequency identification (M-RFID), and/or othercommunication protocols. Other user devices 106 and supervisor devices110 may be general purpose personal computers or special-purposecomputing devices including, by way of example, personal computers,laptop computers, workstation computers, projection devices, andinteractive room display systems. Additionally, user devices 106 andsupervisor devices 110 may be any other electronic devices, such asthin-client computers, Internet-enabled gaming systems, business or homeappliances, and/or personal messaging devices, capable of communicatingover network(s) 120.

In different contexts of content distribution networks 100, user devices106 and supervisor devices 110 may correspond to different types ofspecialized devices, for example, student devices and teacher devices inan educational network, employee devices and presentation devices in acompany network, different gaming devices in a gaming network, etc. Insome embodiments, user devices 106 and supervisor devices 110 mayoperate in the same physical location 107, such as a classroom orconference room. In such cases, the devices may contain components thatsupport direct communications with other nearby devices, such as awireless transceivers and wireless communications interfaces, Ethernetsockets or other Local Area Network (LAN) interfaces, etc. In otherimplementations, the user devices 106 and supervisor devices 110 neednot be used at the same location 107, but may be used in remotegeographic locations in which each user device 106 and supervisor device110 may use security features and/or specialized hardware (e.g.,hardware-accelerated SSL and HTTPS, WS-Security, firewalls, etc.) tocommunicate with the content management server 102 and/or other remotelylocated user devices 106. Additionally, different user devices 106 andsupervisor devices 110 may be assigned different designated roles, suchas presenter devices, teacher devices, administrator devices, or thelike, and in such cases the different devices may be provided withadditional hardware and/or software components to provide content andsupport user capabilities not available to the other devices.

The content distribution network 100 also may include a privacy server108 that maintains private user information at the privacy server 108while using applications or services hosted on other servers. Forexample, the privacy server 108 may be used to maintain private data ofa user within one jurisdiction even though the user is accessing anapplication hosted on a server (e.g., the content management server 102)located outside the jurisdiction. In such cases, the privacy server 108may intercept communications between a user device 106 or supervisordevice 110 and other devices that include private user information. Theprivacy server 108 may create a token or identifier that does notdisclose the private information and may use the token or identifierwhen communicating with the other servers and systems, instead of usingthe user's private information.

As illustrated in FIG. 1, the content management server 102 may be incommunication with one or more additional servers, such as a contentserver 112, a user data server 112, and/or an administrator server 116.Each of these servers may include some or all of the same physical andlogical components as the content management server(s) 102, and in somecases, the hardware and software components of these servers 112-116 maybe incorporated into the content management server(s) 102, rather thanbeing implemented as separate computer servers.

Content server 112 may include hardware and software components togenerate, store, and maintain the content resources for distribution touser devices 106 and other devices in the network 100. For example, incontent distribution networks 100 used for professional training andeducational purposes, content server 112 may include data stores oftraining materials, presentations, interactive programs and simulations,course models, course outlines, and various training interfaces thatcorrespond to different materials and/or different types of user devices106. In content distribution networks 100 used for media distribution,interactive gaming, and the like, a content server 112 may include mediacontent files such as music, movies, television programming, games, andadvertisements.

User data server 114 may include hardware and software components thatstore and process data for multiple users relating to each user'sactivities and usage of the content distribution network 100. Forexample, the content management server 102 may record and track eachuser's system usage, including their user device 106, content resourcesaccessed, and interactions with other user devices 106. This data may bestored and processed by the user data server 114, to support usertracking and analysis features. For instance, in the professionaltraining and educational contexts, the user data server 114 may storeand analyze each user's training materials viewed, presentationsattended, courses completed, interactions, evaluation results, and thelike. The user data server 114 may also include a repository foruser-generated material, such as evaluations and tests completed byusers, and documents and assignments prepared by users. In the contextof media distribution and interactive gaming, the user data server 114may store and process resource access data for multiple users (e.g.,content titles accessed, access times, data usage amounts, gaminghistories, user devices and device types, etc.).

Administrator server 116 may include hardware and software components toinitiate various administrative functions at the content managementserver 102 and other components within the content distribution network100. For example, the administrator server 116 may monitor device statusand performance for the various servers, data stores, and/or userdevices 106 in the content distribution network 100. When necessary, theadministrator server 116 may add or remove devices from the network 100,and perform device maintenance such as providing software updates to thedevices in the network 100. Various administrative tools on theadministrator server 116 may allow authorized users to set user accesspermissions to various content resources, monitor resource usage byusers and devices 106, and perform analyses and generate reports onspecific network users and/or devices (e.g., resource usage trackingreports, training evaluations, etc.).

The content distribution network 100 may include one or morecommunication networks 120. Although only a single network 120 isidentified in FIG. 1, the content distribution network 100 may includeany number of different communication networks between any of thecomputer servers and devices shown in FIG. 1 and/or other devicesdescribed herein. Communication networks 120 may enable communicationbetween the various computing devices, servers, and other components ofthe content distribution network 100. As discussed below, variousimplementations of content distribution networks 100 may employdifferent types of networks 120, for example, computer networks,telecommunications networks, wireless networks, and/or any combinationof these and/or other networks.

With reference to FIG. 2, an illustrative distributed computingenvironment 200 is shown including a computer server 202, four clientcomputing devices 206, and other components that may implement certainembodiments and features described herein. In some embodiments, theserver 202 may correspond to the content management server 102 discussedabove in FIG. 1, and the client computing devices 206 may correspond tothe user devices 106. However, the computing environment 200 illustratedin FIG. 2 may correspond to any other combination of devices and serversconfigured to implement a client-server model or other distributedcomputing architecture.

Client devices 206 may be configured to receive and execute clientapplications over one or more networks 220. Such client applications maybe web browser based applications and/or standalone softwareapplications, such as mobile device applications. Server 202 may becommunicatively coupled with the client devices 206 via one or morecommunication networks 220. Client devices 206 may receive clientapplications from server 202 or from other application providers (e.g.,public or private application stores). Server 202 may be configured torun one or more server software applications or services, for example,web-based or cloud-based services, to support content distribution andinteraction with client devices 206. Users operating client devices 206may in turn utilize one or more client applications (e.g., virtualclient applications) to interact with server 202 to utilize the servicesprovided by these components.

Various different subsystems and/or components 204 may be implemented onserver 202. Users operating the client devices 206 may initiate one ormore client applications to use services provided by these subsystemsand components. The subsystems and components within the server 202 andclient devices 206 may be implemented in hardware, firmware, software,or combinations thereof. Various different system configurations arepossible in different distributed computing systems 200 and contentdistribution networks 100. The embodiment shown in FIG. 2 is thus oneexample of a distributed computing system and is not intended to belimiting.

Although exemplary computing environment 200 is shown with four clientcomputing devices 206, any number of client computing devices may besupported. Other devices, such as specialized sensor devices, etc., mayinteract with client devices 206 and/or server 202.

As shown in FIG. 2, various security and integration components 208 maybe used to send and manage communications between the server 202 anduser devices 206 over one or more communication networks 220. Thesecurity and integration components 208 may include separate servers,such as web servers and/or authentication servers, and/or specializednetworking components, such as firewalls, routers, gateways, loadbalancers, and the like. In some cases, the security and integrationcomponents 208 may correspond to a set of dedicated hardware and/orsoftware operating at the same physical location and under the controlof same entities as server 202. For example, components 208 may includeone or more dedicated web servers and network hardware in a datacenteror a cloud infrastructure. In other examples, the security andintegration components 208 may correspond to separate hardware andsoftware components which may be operated at a separate physicallocation and/or by a separate entity.

Security and integration components 208 may implement various securityfeatures for data transmission and storage, such as authenticating usersand restricting access to unknown or unauthorized users. In variousimplementations, security and integration components 208 may provide,for example, a file-based integration scheme or a service-basedintegration scheme for transmitting data between the various devices inthe content distribution network 100. Security and integrationcomponents 208 also may use secure data transmission protocols and/orencryption for data transfers, for example, File Transfer Protocol(FTP), Secure File Transfer Protocol (SFTP), and/or Pretty Good Privacy(PGP) encryption.

In some embodiments, one or more web services may be implemented withinthe security and integration components 208 and/or elsewhere within thecontent distribution network 100. Such web services, includingcross-domain and/or cross-platform web services, may be developed forenterprise use in accordance with various web service standards, such asthe Web Service Interoperability (WS-I) guidelines. For example, someweb services may use the Secure Sockets Layer (SSL) or Transport LayerSecurity (TLS) protocol to provide secure connections between the server202 and user devices 206. SSL or TLS may use HTTP or HTTPS to provideauthentication and confidentiality. In other examples, web services maybe implemented using the WS-Security standard, which provides for secureSOAP messages using XML encryption. In other examples, the security andintegration components 208 may include specialized hardware forproviding secure web services. For example, security and integrationcomponents 208 may include secure network appliances having built-infeatures such as hardware-accelerated SSL and HTTPS, WS-Security, andfirewalls. Such specialized hardware may be installed and configured infront of any web servers, so that any external devices may communicatedirectly with the specialized hardware.

Communication network(s) 220 may be any type of network familiar tothose skilled in the art that can support data communications using anyof a variety of commercially-available protocols, including withoutlimitation, TCP/IP (transmission control protocol/Internet protocol),SNA (systems network architecture), IPX (Internet packet exchange),Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols,Hyper Text Transfer Protocol (HTTP) and Secure Hyper Text TransferProtocol (HTTPS), and the like. Merely by way of example, network(s) 220may be local area networks (LAN), such as one based on Ethernet,Token-Ring and/or the like. Network(s) 220 also may be wide-areanetworks, such as the Internet. Networks 220 may includetelecommunication networks such as a public switched telephone networks(PSTNs), or virtual networks such as an intranet or an extranet.Infrared and wireless networks (e.g., using the Institute of Electricaland Electronics (IEEE) 802.11 protocol suite or other wirelessprotocols) also may be included in networks 220.

Computing environment 200 also may include one or more data stores 210and/or back-end servers 212. In certain examples, the data stores 210may correspond to data store server(s) 104 discussed above in FIG. 1,and back-end servers 212 may correspond to the various back-end servers112-116. Data stores 210 and servers 212 may reside in the samedatacenter or may operate at a remote location from server 202. In somecases, one or more data stores 210 may reside on a non-transitorystorage medium within the server 202. Other data stores 210 and back-endservers 212 may be remote from server 202 and configured to communicatewith server 202 via one or more networks 220. In certain embodiments,data stores 210 and back-end servers 212 may reside in a storage-areanetwork (SAN).

With reference to FIG. 3, an illustrative set of data stores and/or datastore servers is shown, corresponding to the data store servers 104 ofthe content distribution network 100 discussed above in FIG. 1. One ormore individual data stores 301-309 may reside in storage on a singlecomputer server 104 (or a single server farm or cluster) under thecontrol of a single entity, or may reside on separate servers operatedby different entities and/or at remote locations. In some embodiments,data stores 301-309 may be accessed by the content management server 102and/or other devices and servers within the network 100 (e.g., userdevices 106, supervisor devices 110, administrator servers 116, etc.).Access to one or more of the data stores 301-309 may be limited ordenied based on the processes, user credentials, and/or devicesattempting to interact with the data store.

The paragraphs below describe examples of specific data stores that maybe implemented within some embodiments of a content distribution network100. It should be understood that the below descriptions of data stores301-309, including their functionality and types of data stored therein,are illustrative and non-limiting. Data stores server architecture,design, and the execution of specific data stores 301-309 may depend onthe context, size, and functional requirements of a content distributionnetwork 100. For example, in content distribution systems 100 used forprofessional training and educational purposes, separate databases orfile-based storage systems may be implemented in data store server(s)104 to store trainee and/or student data, trainer and/or professor data,training module data and content descriptions, training results,evaluation data, and the like. In contrast, in content distributionsystems 100 used for media distribution from content providers tosubscribers, separate data stores may be implemented in data storesserver(s) 104 to store listings of available content titles anddescriptions, content title usage statistics, subscriber profiles,account data, payment data, network usage statistics, etc.

A user profile data store 301 may include information relating to theend users within the content distribution network 100. This informationmay include user characteristics such as the user names, accesscredentials (e.g., logins and passwords), user preferences, andinformation relating to any previous user interactions within thecontent distribution network 100 (e.g., requested content, postedcontent, content modules completed, training scores or evaluations,other associated users, etc.).

An accounts data store 302 may generate and store account data fordifferent users in various roles within the content distribution network100. For example, accounts may be created in an accounts data store 302for individual end users, supervisors, administrator users, and entitiessuch as companies or educational institutions. Account data may includeaccount types, current account status, account characteristics, and anyparameters, limits, restrictions associated with the accounts.

A content library data store 303 may include information describing theindividual content items (or content resources) available via thecontent distribution network 100. In some embodiments, the library datastore 303 may include metadata, properties, and other characteristicsassociated with the content resources stored in the content server 112.Such data may identify one or more aspects or content attributes of theassociated content resources, for example, subject matter, access level,or skill level of the content resources, license attributes of thecontent resources (e.g., any limitations and/or restrictions on thelicensable use and/or distribution of the content resource), priceattributes of the content resources (e.g., a price and/or pricestructure for determining a payment amount for use or distribution ofthe content resource), rating attributes for the content resources(e.g., data indicating the evaluation or effectiveness of the contentresource), and the like. In some embodiments, the library data store 303may be configured to allow updating of content metadata or properties,and to allow the addition and/or removal of information relating to thecontent resources.

A pricing data store 304 may include pricing information and/or pricingstructures for determining payment amounts for providing access to thecontent distribution network 100 and/or the individual content resourceswithin the network 100. In some cases, pricing may be determined basedon a user's access to the content distribution network 100, for example,a time-based subscription fee, or pricing based on network usage and. Inother cases, pricing may be tied to specific content resources. Certaincontent resources may have associated pricing information, whereas otherpricing determinations may be based on the resources accessed, theprofiles and/or accounts of the user, and the desired level of access(e.g., duration of access, network speed, etc.). Additionally, thepricing data store 304 may include information relating to compilationpricing for groups of content resources, such as group prices and/orprice structures for groupings of resources.

A license data store 305 may include information relating to licensesand/or licensing of the content resources within the contentdistribution network 100. For example, the license data store 305 mayidentify licenses and licensing terms for individual content resourcesand/or compilations of content resources in the content server 112, therights holders for the content resources, and/or common or large-scaleright holder information such as contact information for rights holdersof content not included in the content server 112.

A content access data store 306 may include access rights and securityinformation for the content distribution network 100 and specificcontent resources. For example, the content access data store 306 mayinclude login information (e.g., user identifiers, logins, passwords,etc.) that can be verified during user login attempts to the network100. The content access data store 306 also may be used to storeassigned user roles and/or user levels of access. For example, a user'saccess level may correspond to the sets of content resources and/or theclient or server applications that the user is permitted to access.Certain users may be permitted or denied access to certain applicationsand resources based on their subscription level, training program,course/grade level, etc. Certain users may have supervisory access overone or more end users, allowing the supervisor to access all or portionsof the end user's content, activities, evaluations, etc. Additionally,certain users may have administrative access over some users and/or someapplications in the content management network 100, allowing such usersto add and remove user accounts, modify user access permissions, performmaintenance updates on software and servers, etc.

A source data store 307 may include information relating to the sourceof the content resources available via the content distribution network.For example, a source data store 307 may identify the authors andoriginating devices of content resources, previous pieces of data and/orgroups of data originating from the same authors or originating devices,and the like.

An evaluation data store 308 may include information used to direct theevaluation of users and content resources in the content managementnetwork 100. In some embodiments, the evaluation data store 308 maycontain, for example, the analysis criteria and the analysis guidelinesfor evaluating users (e.g., trainees/students, gaming users, mediacontent consumers, etc.) and/or for evaluating the content resources inthe network 100. The evaluation data store 308 also may includeinformation relating to evaluation processing tasks, for example, theidentification of users and user devices 106 that have received certaincontent resources or accessed certain applications, the status ofevaluations or evaluation histories for content resources, users, orapplications, and the like. Evaluation criteria may be stored in theevaluation data store 308 including data and/or instructions in the formof one or several electronic rubrics or scoring guides for use in theevaluation of the content, users, or applications. The evaluation datastore 308 also may include past evaluations and/or evaluation analysesfor users, content, and applications, including relative rankings,characterizations, explanations, and the like.

In addition to the illustrative data stores described above, data storeserver(s) 104 (e.g., database servers, file-based storage servers, etc.)may include one or more external data aggregators 309. External dataaggregators 309 may include third-party data sources accessible to thecontent management network 100, but not maintained by the contentmanagement network 100. External data aggregators 309 may include anyelectronic information source relating to the users, content resources,or applications of the content distribution network 100. For example,external data aggregators 309 may be third-party data stores containingdemographic data, education related data, consumer sales data, healthrelated data, and the like. Illustrative external data aggregators 309may include, for example, social networking web servers, public recordsdata stores, learning management systems, educational institutionservers, business servers, consumer sales data stores, medical recorddata stores, etc. Data retrieved from various external data aggregators309 may be used to verify and update user account information, suggestuser content, and perform user and content evaluations.

With reference now to FIG. 4, a block diagram is shown illustrating anembodiment of one or more content management servers 102 within acontent distribution network 100. As discussed above, content managementserver(s) 102 may include various server hardware and softwarecomponents that manage the content resources within the contentdistribution network 100 and provide interactive and adaptive content tousers on various user devices 106. For example, content managementserver(s) 102 may provide instructions to and receive information fromthe other devices within the content distribution network 100, in orderto manage and transmit content resources, user data, and server orclient applications executing within the network 100.

A content management server 102 may include a content customizationsystem 402. The content customization system 402 may be implementedusing dedicated hardware within the content distribution network 100(e.g., a content customization server 402), or using designated hardwareand software resources within a shared content management server 102. Insome embodiments, the content customization system 402 may adjust theselection and adaptive capabilities of content resources to match theneeds and desires of the users receiving the content. For example, thecontent customization system 402 may query various data stores andservers 104 to retrieve user information, such as user preferences andcharacteristics (e.g., from a user profile data store 301), user accessrestrictions to content recourses (e.g., from a content access datastore 306), previous user results and content evaluations (e.g., from anevaluation data store 308), and the like. Based on the retrievedinformation from data stores 104 and other data sources, the contentcustomization system 402 may modify content resources for individualusers.

A content management server 102 also may include a user managementsystem 404. The user management system 404 may be implemented usingdedicated hardware within the content distribution network 100 (e.g., auser management server 404), or using designated hardware and softwareresources within a shared content management server 102. In someembodiments, the user management system 404 may monitor the progress ofusers through various types of content resources and groups, such asmedia compilations, courses or curriculums in training or educationalcontexts, interactive gaming environments, and the like. For example,the user management system 404 may query one or more databases and/ordata store servers 104 to retrieve user data such as associated contentcompilations or programs, content completion status, user goals,results, and the like.

A content management server 102 also may include an evaluation system406. The evaluation system 406 may be implemented using dedicatedhardware within the content distribution network 100 (e.g., anevaluation server 406), or using designated hardware and softwareresources within a shared content management server 102. The evaluationsystem 406 may be configured to receive and analyze information fromuser devices 106. For example, various ratings of content resourcessubmitted by users may be compiled and analyzed, and then stored in adata store (e.g., a content library data store 303 and/or evaluationdata store 308) associated with the content. In some embodiments, theevaluation server 406 may analyze the information to determine theeffectiveness or appropriateness of content resources with, for example,a subject matter, an age group, a skill level, or the like. In someembodiments, the evaluation system 406 may provide updates to thecontent customization system 402 or the user management system 404, withthe attributes of one or more content resources or groups of resourceswithin the network 100. The evaluation system 406 also may receive andanalyze user evaluation data from user devices 106, supervisor devices110, and administrator servers 116, etc. For instance, evaluation system406 may receive, aggregate, and analyze user evaluation data fordifferent types of users (e.g., end users, supervisors, administrators,etc.) in different contexts (e.g., media consumer ratings, trainee orstudent comprehension levels, teacher effectiveness levels, gamer skilllevels, etc.).

A content management server 102 also may include a content deliverysystem 408. The content delivery system 408 may be implemented usingdedicated hardware within the content distribution network 100 (e.g., acontent delivery server 408), or using designated hardware and softwareresources within a shared content management server 102. The contentdelivery system 408 may receive content resources from the contentcustomization system 402 and/or from the user management system 404, andprovide the resources to user devices 106. The content delivery system408 may determine the appropriate presentation format for the contentresources based on the user characteristics and preferences, and/or thedevice capabilities of user devices 106. If needed, the content deliverysystem 408 may convert the content resources to the appropriatepresentation format and/or compress the content before transmission. Insome embodiments, the content delivery system 408 may also determine theappropriate transmission media and communication protocols fortransmission of the content resources.

In some embodiments, the content delivery system 408 may includespecialized security and integration hardware 410, along withcorresponding software components to implement the appropriate securityfeatures content transmission and storage, to provide the supportednetwork and client access models, and to support the performance andscalability requirements of the network 100. The security andintegration layer 410 may include some or all of the security andintegration components 208 discussed above in FIG. 2, and may controlthe transmission of content resources and other data, as well as thereceipt of requests and content interactions, to and from the userdevices 106, supervisor devices 110, administrative servers 116, andother devices in the network 100.

With reference now to FIG. 5, a block diagram of an illustrativecomputer system is shown. The system 500 may correspond to any of thecomputing devices or servers of the content distribution network 100described above, or any other computing devices described herein. Inthis example, computer system 500 includes processing units 504 thatcommunicate with a number of peripheral subsystems via a bus subsystem502. These peripheral subsystems include, for example, a storagesubsystem 510, an I/O subsystem 526, and a communications subsystem 532.

Bus subsystem 502 provides a mechanism for letting the variouscomponents and subsystems of computer system 500 communicate with eachother as intended. Although bus subsystem 502 is shown schematically asa single bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 502 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 504, which may be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 500. One or more processors,including single core and/or multicore processors, may be included inprocessing unit 504. As shown in the figure, processing unit 504 may beimplemented as one or more independent processing units 506 and/or 508with single or multicore processors and processor caches included ineach processing unit. In other embodiments, processing unit 504 may alsobe implemented as a quad-core processing unit or larger multicoredesigns (e.g., hexa-core processors, octo-core processors, ten-coreprocessors, or greater.

Processing unit 504 may execute a variety of software processes embodiedin program code, and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed can be resident in processor(s) 504 and/or instorage subsystem 510. In some embodiments, computer system 500 mayinclude one or more specialized processors, such as digital signalprocessors (DSPs), outboard processors, graphics processors,application-specific processors, and/or the like.

I/O subsystem 526 may include device controllers 528 for one or moreuser interface input devices and/or user interface output devices 530.User interface input and output devices 530 may be integral with thecomputer system 500 (e.g., integrated audio/video systems, and/ortouchscreen displays), or may be separate peripheral devices which areattachable/detachable from the computer system 500.

Input devices 530 may include a keyboard, pointing devices such as amouse or trackball, a touchpad or touch screen incorporated into adisplay, a scroll wheel, a click wheel, a dial, a button, a switch, akeypad, audio input devices with voice command recognition systems,microphones, and other types of input devices. Input devices 530 mayalso include three dimensional (3D) mice, joysticks or pointing sticks,gamepads and graphic tablets, and audio/visual devices such as speakers,digital cameras, digital camcorders, portable media players, webcams,image scanners, fingerprint scanners, barcode reader 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices. Additionalinput devices 530 may include, for example, motion sensing and/orgesture recognition devices that enable users to control and interactwith an input device through a natural user interface using gestures andspoken commands, eye gesture recognition devices that detect eyeactivity from users and transform the eye gestures as input into aninput device, voice recognition sensing devices that enable users tointeract with voice recognition systems through voice commands, medicalimaging input devices, MIDI keyboards, digital musical instruments, andthe like.

Output devices 530 may include one or more display subsystems, indicatorlights, or non-visual displays such as audio output devices, etc.Display subsystems may include, for example, cathode ray tube (CRT)displays, flat-panel devices, such as those using a liquid crystaldisplay (LCD) or plasma display, projection devices, touch screens, andthe like. In general, use of the term “output device” is intended toinclude all possible types of devices and mechanisms for outputtinginformation from computer system 500 to a user or other computer. Forexample, output devices 530 may include, without limitation, a varietyof display devices that visually convey text, graphics and audio/videoinformation such as monitors, printers, speakers, headphones, automotivenavigation systems, plotters, voice output devices, and modems.

Computer system 500 may comprise one or more storage subsystems 510,comprising hardware and software components used for storing data andprogram instructions, such as system memory 518 and computer-readablestorage media 516. The system memory 518 and/or computer-readablestorage media 516 may store program instructions that are loadable andexecutable on processing units 504, as well as data generated during theexecution of these programs.

Depending on the configuration and type of computer system 500, systemmemory 318 may be stored in volatile memory (such as random accessmemory (RAM) 512) and/or in non-volatile storage drives 514 (such asread-only memory (ROM), flash memory, etc.) The RAM 512 may contain dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 504. In someimplementations, system memory 518 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 500, such as duringstart-up, may typically be stored in the non-volatile storage drives514. By way of example, and not limitation, system memory 518 mayinclude application programs 520, such as client applications, Webbrowsers, mid-tier applications, server applications, etc., program data522, and an operating system 524.

Storage subsystem 510 also may provide one or more tangiblecomputer-readable storage media 516 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 510. These software modules or instructions may beexecuted by processing units 504. Storage subsystem 510 may also providea repository for storing data used in accordance with the presentinvention.

Storage subsystem 300 may also include a computer-readable storage mediareader that can further be connected to computer-readable storage media516. Together and, optionally, in combination with system memory 518,computer-readable storage media 516 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 516 containing program code, or portionsof program code, may include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computer system 500.

By way of example, computer-readable storage media 516 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 516 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 516 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 500.

Communications subsystem 532 may provide a communication interface fromcomputer system 500 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 5, thecommunications subsystem 532 may include, for example, one or morenetwork interface controllers (NICs) 534, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 536, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 532 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, FireWire® interfaces, USB® interfaces, and the like.Communications subsystem 536 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

The various physical components of the communications subsystem 532 maybe detachable components coupled to the computer system 500 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 500. Communicationssubsystem 532 also may be implemented in whole or in part by software.

In some embodiments, communications subsystem 532 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 500. For example,communications subsystem 532 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators 309). Additionally, communications subsystem 532may be configured to receive data in the form of continuous datastreams, which may include event streams of real-time events and/orevent updates (e.g., sensor data applications, financial tickers,network performance measuring tools, clickstream analysis tools,automobile traffic monitoring, etc.). Communications subsystem 532 mayoutput such structured and/or unstructured data feeds, event streams,event updates, and the like to one or more data stores 104 that may bein communication with one or more streaming data source computerscoupled to computer system 500.

Due to the ever-changing nature of computers and networks, thedescription of computer system 500 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

With reference now to FIG. 6, a block diagram is shown illustrating anexample of a mathematical expression conversion system 600. In variousimplementations, an expression conversion system 600 may include one ormore expression input devices 610, refreshable braille display devices630, and/or visual display devices 640, each of which may include or maybe configured to communicate with an expression converter 620. Asdiscussed below, the devices and components discussed in reference tosystem 600 may receive and parse input corresponding to variousdifferent types of mathematical expressions, perform one or moreconversions on the received mathematical expressions, and then generatebraille and/or visual output representing the mathematical expressionsvia the refreshable braille display devices 630 and/or visual displaydevices 640. More specifically, in various embodiments, expression inputdevices 610, refreshable braille display devices 630, and visual displaydevices 640 may be configured to receive, parse, and transmit datacorresponding to inputs of mathematical expressions, as well as receiveand output converted mathematical expressions using braille outputcharacters and/or visual output characters. Expression converters 620may be configured to receive mathematical expression input data, performvarious conversions on the input data, and then output convertedmathematical expressions to various appropriate user devices 410, 430,and 440. Such conversions may include, for example, mathematicalexpression input to content markup conversions, braille character inputto content markup conversions, content markup to braille characteroutput conversions, and content markup to presentation markupconversions.

In order to perform these features and other functionality describedherein, each of the components and sub-components discussed in theexample mathematical expression conversion system 600 may correspond toa single computer server or a complex computing system including acombination of computing devices, storage devices, network components,etc. Each of these components and their respective subcomponents may beimplemented in hardware, software, or a combination thereof. Certainexpression input devices 610, refreshable braille display devices 630,and/or visual display devices 640 may communicate directly with theexpression converter 620, while other such devices may communicate withthe expression converter 620 indirectly via one or more intermediarynetwork components (e.g., routers, gateways, firewalls, etc.) or otherdevices in a content distribution network (e.g., content managementservers 102, content servers 112, etc.). Although the physicalcomponents of communication networks 220 have not been shown in thisfigure so as not to obscure the other elements depicted in the figure,it should be understood that any of the network hardware components andnetwork architecture designs may be implemented in various embodimentsto support communication between the servers, devices, and data storesin the system 600. Additionally, different devices 610, 630, and/or 640may use different networks and networks types to communicate with eachother and with the expression converter 620, including one or moretelecommunications networks, cable networks, satellite networks,cellular networks and other wireless networks, and computer-based IPnetworks, and the like. Further, certain components within mathematicalexpression conversion system 600 may include special purpose hardwaredevices and/or special purpose software, such as those included in I/Osubsystems of expression input devices 610, refreshable braille displaydevices 630, and/or visual display devices 640, in order to receive,parse, render, and display various mathematical expressions usingbraille or visual output characters. Additional special purpose hardwaredevices and/or special purpose software may be included within theprocessing units and memory systems of the expression converter 620and/or devices 610, 630, and 640, in order to parse, store, and convertvarious types of special and non-standard mathematical characters andsymbols, and complex mathematical expressions.

The mathematical expression conversion system 600, and other examplesand embodiments described herein, may be used to receive input datacorresponding to mathematical expressions, parse and convert thereceived expressions, and output braille and visual characterrepresentations of the expressions on various display devices. As usedherein, a mathematical expression may be any expression containing oneor more mathematical symbols, including numbers, variables, operators,and functions, as well as grouping symbols and any other mathematicalsymbol. Different types of mathematical expressions that may bereceived, converted, generated, and/or output in the various embodimentsdescribed herein may include arithmetic expressions, algebraicexpressions, polynomial expressions, closed-form expressions, and thelike. Examples of specific mathematical expressions may include anycombination of one or more constants, variables, arithmetic operators,factorials, exponents, roots, logarithms, trigonometric functions,integrals, differentials, and the like. Moreover, it should beunderstood that these specific examples and the other examples ofmathematical expressions described herein are illustrative only andnon-limiting, and that the techniques described herein may be used toreceive, parse, convert, and generate braille and/or visual output forany mathematical expression.

In some embodiments, mathematical expression conversion systems 600 maybe integrated within, or configured to operate in collaboration with,one or more content distribution networks 100. For example, system 600may be the same as, or may operate within or in collaboration with, anyof the content distribution network (CDNs) 100 described above. Thus,specific implementations of a mathematical expression conversion system600 may include, without limitation, educational and professionaltraining systems and networks, interactive gaming systems and networks,presentation systems and networks, collaborative working systems andnetworks, websites and other Internet-based systems and networks, etc.In such cases, expression converters 620 may be implemented within oneor more content servers 112, content management servers 102, and/or datastore servers 104, and expression input devices 610, refreshable brailledisplay devices 630, and visual display devices 640 correspond to theuser devices 106 and 110 described above in reference to CDN 100. Inother examples, expression converter 620 may be implemented using one ormore computer servers, and other specialized hardware and softwarecomponents, separately from other CDN components such as content servers112, content management servers 102, data store servers 104, and thelike. In these examples, the expression converter 620 may be configuredto communicate directly with devices 610, 630, and 640, or indirectlythrough content management servers 102 and/or other components andcommunications networks of a CDN 100.

Expression input devices 610 may include any computing device configuredto receive user input corresponding to a mathematical expression. Thus,expression input devices 610 may include desktop and laptop computers,smartphones, tablet computers, mobile devices, and the like. As shown inthis example, certain expression input devices 610 may include one ormore input components 611 and/or output components 612, which may beperipheral devices which are attachable/detachable from the from thedevice 610, or may be integrated within the device 610. Input components611 may include, for example, keyboards, mouses, styluses, touchscreendisplays, cameras, microphones, etc. Output components 612 may includevarious display screens, audio and/or video systems, etc. In some cases,expression input devices 610 may include specialized hardware componentsand/or software components for inputting and outputting mathematicalexpressions. For instance, an expression input device 610 may include anintegrated or peripheral specialized mathematics keyboard 611 tofacilitate the inputting of special and non-standard mathematicalsymbols. Additionally, specialized display screens 612 and other outputsmay be used in some cases to render and display special and non-standardmathematical symbols and expressions. In certain embodiments, expressioninput devices 610 cases may include combinations of specialized softwareand hardware components used for inputting and outputting mathematicalexpressions. For example, a touchscreen display and/or stylus pen may becombined with software functionality used to identify and distinguishmathematical symbols and combinations of symbols in expressions. Inother cases, sequences and/or combinations of keys may be depressed on astandard keyboard 411 or touchscreen keyboard, such that each unique keysequence or combination may represent a different mathematical symbol. Aspecialized software-based math input display panel also may be renderedon a display screen 612, and the user may identify and select a desiredsymbol using a mouse, keyboard, or touchscreen capability, etc. In stillother cases, microphones and/or cameras may be used along with voice orgesture recognition and control software to allow users to inputmathematical symbols and expressions via an expression input device 610.

Mathematical expression conversion systems 600 also may include variousdisplay devices configured to output data corresponding to mathematicalexpressions, including one or more refreshable braille display devices630 and one or more visual display devices 640. In some embodiments,expression input devices 610 may be similar or identical to displaydevices 630 and/or 640, in that such devices may include both inputcomponents for receiving mathematical expression input data and outputcomponents for displaying mathematical expression output data. In fact,designated expression input devices 610 may be optional in some systems600, and one or more of the display devices 630 and/or 640 may be usedfor inputting and outputting mathematical expression data. Additionally,although these examples include both refreshable braille display devices630 and visual display devices 640, other examples may include either aset of refreshable braille display devices 630 or a set of visualdisplay devices 640, but not both. As discussed below, expressionconverters and/or conversion processes may be used even in cases whenonly one type of display device (e.g., 630 or 640) is used within thesystem 600.

Refreshable braille display devices 630 may include one or moreelectromechanical braille cells configured to display braille charactersfor braille readers. In some cases, a plurality of electromechanicalbraille cells may each include a plurality of output dots (e.g.,round-tipped pins) which may be raised and lowered based on the voltagesapplied to the electromechanical braille cell. Different combinations ofoutput dots may be raised and lowered to represent different characters.As discussed below, certain common mathematical symbols such as letters,numbers, and arithmetic operators may be represented using a singlebraille cell, while multiple braille cells may be required to representother mathematical symbols such as special and non-standard symbols.Illustrative examples of refreshable braille display devices 630 mayinclude, without limitation, the BRAILLE SENSE U2 NOTETAKER by HIMSINC., or the BRAILLIANT BRAILLE DISPLAY by HUMANWARE, among others.

In addition to the panels of electromechanical braille cells 632,certain braille display devices 630 may include a keypad 631 for entryof mathematical symbols and expressions. In some cases, keypads 631 maycorrespond to a standard keyboard, optionally including braille dotsaffixed to some or all of the individual keys to aid visually impairedusers. In other cases, keypads 631 may include specialized braillekeyboards (e.g., Perkins style, 6-key or 8-key chorded keyboards usedfor braille typing). Additionally, in some embodiments, one or morebraille display devices 630 may include additional hardware and softwarefor receiving mathematical expression input and displaying mathematicalexpression output for visually impaired users, such as microphones andvoice recognition and control functionality for receiving/processingmathematical symbols and expressions, and device speakers and speechsynthesizers for outputting mathematical symbols and expressions.

Visual display devices 640 may include any computing devices configuredto receive and display mathematical expression output. As previouslynoted, visual display devices 640 may be similar or identical to theexpression input devices 610, and thus may include desktop and laptopcomputers, smartphones, tablet computers, mobile devices, and any othercomputing device capable of displaying mathematical symbols andexpressions. As shown in this example, certain visual display devices640 may include one or more input components 641 and/or outputcomponents 642, which may include peripheral devices or I/O componentsintegrated within the device. For example, output components 642 maycorrespond to display screens for displaying mathematical expressions.In some cases, visual display devices 640 may include standard computerdisplay screens 642, while in other cases the display screens 642 may bespecialized for rendering and displaying mathematical symbols andexpressions. For instance, displaying of certain mathematical symbolsand expressions may require (or may preferably include) displaycapabilities such as a minimum screen size, minimum display resolution,color display, graphics or video capabilities, 3D display capabilities,and the like. Additionally, visual display devices 640 including suchspecialized display screens 642 may include corresponding specializedsoftware components and underlying hardware for rendering and outputtingto the display screen 642, such as specialized graphics components(e.g., graphics processors, graphics drivers, 3D graphics support,etc.), mathematical symbol libraries, mathematical expression renderingsoftware packages, and the like.

In shown in this example, expression converter 620 may communicate withexpression input devices 610 and one or more braille display devices 630and/or visual display devices 640 via communication networks 220. Asshown in this example, an expression converter 620 may be implemented asa server, device, or other standalone hardware and software systemwithin a mathematical expression conversion system 600. In otherexamples, an expression converter 620 may be integrated within one ormore servers or other components of an associated CDN 100. For example,an expression converter 620 may be implemented as specialized softwarewithin a content management server 102 or other CDN component, and mayleverage much of the existing hardware and software infrastructure ofthe CDN 100 to convert mathematical expressions for devices 610, 630,and 640 (which may correspond to user devices 106 and supervisor devices110, etc.) within an educational or professional training CDN 100, amathematical presentation and collaboration CDN 100, an interactivemath-based gaming CDN 100, etc.

Expression converter 620 may include multiple different convertersand/or conversion processes for performing the various mathematicalexpression conversions described herein. For example, expressionconverter 620 may include one or more of a math input-to-content markupconverter 621, a content markup-to-braille converter 622, a contentmarkup-to-presentation markup converter 623, and a braille-to-contentmarkup converter 624. The functionality of converters 621-624 isdescribed below in reference to FIGS. 7-9. In various embodiments,converters 621-624 may be implemented as separate converters usingdedicated hardware resources and/or executable software components.Alternatively or additionally, some or all of the converters 621-624 maybe combined to use the same software executable and/or underlyinghardware resources. In any such embodiments, each converter 621-624 maymaintain and apply a specific set of mathematical expression conversionrules that is unique to the individual converter 621-624 and/orconversion processes applied. As discussed below in more detail, theseexpression conversion rules may include character mapping rules as wellas special encoding and decoding rules used specifically formathematical expressions that do not correspond to character mappingrules. Different sets of special encoding and decoding rules may bestored and applied by each converter 621-624 and/or conversion process,and may be used to provide uniformity of presentation and to reduce (oreven eliminate) ambiguous mathematical expressions during the conversionand braille and/or visual display processes. The various conversionrules for mathematical expressions used by converters 621-624 may bestored within the memory of the converters 621-624 themselves, or withinan external data store 625. In some embodiments, storing some or allsuch conversion rule within an external data store 625 may allow for therules to be updated dynamically and seamlessly without affecting theexecution of the converters 621-624.

Additionally, in some embodiments, the expression converter 620 mightnot be implemented separately from the expression input devices 610,refreshable braille display devices 630, and/or visual display devices640, but instead may be implemented within these devices leveraging thevarious hardware and software components therein. Thus, implementing theexpression converter 620 as a separate server or device as shown in FIG.6, and/or within any CDN server-side component (e.g., CMS 102, contentserver 112, administrative server 116, etc.) may be optional in suchembodiments. Instead, each of the expression input devices 610,refreshable braille display devices 630, and/or visual display devices640 may include one or more of the converter components 621-624. Forinstance, refreshable braille display devices 430 may include a contentmarkup-to-braille converter 622 and a braille-to-content markupconverter 624. Similarly, expression input devices 610 and/or visualdisplay devices 440 may include a math input-to-content markup converter621, and a content markup-to-presentation markup converter 623. Forvisual display devices 440 that support braille display capabilities(see, e.g., FIG. 12-14), such visual display devices 440 may alsoinclude braille converters 622 and 624.

Referring now to FIG. 7, a flow diagram is shown illustrating a processof converting mathematical expression data received from an input deviceto braille output and/or visual display output. As described below, thesteps in this process may be performed by one or more components in themathematical expression conversion systems 600 described above, such asan expression converter 620 and/or various input and output devices 610,630, and 640. However, it should be understood that the various featuresand processes described herein, including receiving input datacorresponding to mathematical expressions, performing conversionprocesses on the mathematical expression input data, determining outputdevices, and transmitting braille output and/or visual output to theappropriate output devices need not be limited to the specific systemsand hardware implementations described above in FIGS. 1-6.

In step 701, mathematical expression input data is received, forexample, by an expression converter 620 within a mathematical expressionconversion system 600. As discussed above, in various differentembodiments, expression converters 620 may be implemented as standaloneservers or devices, integrated within the existing server infrastructureof a CDN 100, and/or integrated within individual user devices such asthe expression input devices 610, refreshable braille display devices630, and/or visual display devices 640 discussed above.

The mathematical expression input data received in step 701 may consistof a single character corresponding to any mathematical symbol (e.g., anumber, variable, operator, functions, grouping symbol, etc.) or maycomprise an expression having multiple symbols. In some cases, themathematical expression input data received in step 701 may be receivedin response to a user input of one or more characters via an expressioninput device 610 or other user devices 630 and 640 with inputcapabilities. For example, within an implementation of an educational orprofessional training software system 600, an interactive gaming system600, an online presentation system 600, a collaborative work environment600, etc., a first user (e.g., a presenter, teacher, trainer, etc.) mayinput one or more mathematical symbols into a mathematical userinterface provided on an expression input device 610. In response, theexpression input device 610 may transmit data to the expressionconverter 620 identifying the input characters/symbols. As discussedbelow, this action by the first user (e.g., inputting or updating asingle character of a mathematical expression) may trigger theconversions of the input data and the outputting of the convertedmathematical expressions to visual display devices 640 and/orrefreshable braille display devices 630, which may be performed inreal-time or near real-time in response to the mathematical expressioninput data received in step 701.

In other cases, the mathematical expression input data received in step701 need not be received from an expression input device 610 or otheruser device, but may correspond to data received over a communicationnetwork 220. For example, in some embodiments, the receipt of themathematical expression input data in step 701 may correspond to theexpression converter 620 retrieving a web page or other documentcontaining one or more mathematical expressions from a remote server.Such documents may correspond to any type of content resource that maybe retrieved and consumed within a CDN 100, such as media content,presentations, educational or professional training content, interactivegaming content, web-based content, any of which may be stored by acontent server 112 and/or provided by a content management server 102.In these examples, a server or client system within the CDN 100 maydetermine that the requested/retrieved content resources include one ormore mathematical expressions, and in response may invoke an expressionconverter 620 implemented within a server (e.g., CMS 102 and/or contentserver 112), or within a client device such as user devices 106 and/or110 which may correspond to visual display devices 640 and/orrefreshable braille display devices 630, in order to convert and theoutput the mathematical expression data to the appropriate user devices.

In step 702, the expression converter 620 may execute one or moreconversion processes on the mathematical expression input data receivedin step 701 to convert the mathematical expression to content markup. Asused herein, content markup for a mathematical expression may refer to alanguage or code defining a specified formatting, style, and/or layoutfor representing mathematical expressions based on the underlyingstructure of the expression. Thus, content markup is encoded explicitlybased on the mathematical structure of the expression, rather than basedon any particular visual or oral rendering or representation of theexpression. Content markup may be implemented by directly encoding theexpression tree structure of the mathematical expression, without anydependency on an additional processing or special parsing of theexpression. Thus, a content markup representation of a mathematicalexpression may be entirely non-ambiguous, unlike certain textual orvisual representations of mathematical expression that may be ambiguousin some cases. Examples of different content markup definitions that maybe used in step 702 include any of content markup specifications withinthe Mathematical Markup Language (MathML) editions and/or versionsdefined by the World Wide Web Consortium (W3C), although it should beunderstood that these examples are illustrative only and non-limiting,and that other content markup definitions or specifications may be usedin other examples.

In some embodiments, the conversion process performed in step 702 mayinclude a conversion from ASCII, Unicode (e.g., UTF-8 or UTF-16), oranother visual display format, to content markup. To perform theconversion in step 702, the expression converter 620 may invoke themath-to-content markup converter 621 and/or the applicable expressionconversion rules for math-to-content markup conversion from anexpression conversion rules data store 625. Converter 621 may beconfigured to parse the mathematical expression input, identify thestructural components of the expression (e.g., determine an expressiontree corresponding to the input), and then generate and populate thecorresponding markup elements. Additional conversions that may beperformed in step 702 may include an initial handwriting to ASCII orUnicode conversion, voice-to-text conversion, etc., which may be usedwhen the mathematical expression input data received in step 701corresponds to handwriting or voice data, and so on. Referring brieflyto FIGS. 10A and 11A, two examples of content markup are shown that maybe generated in step 702, based on received input corresponding to thetextual mathematical expression shown above each text box in theseexamples.

In step 703, the expression converter 620 may determine whether or notany refreshable braille display devices 630 are present in the system600 to receive the mathematical expression output. For example, themathematical expression input in step 701 may be received during aninteractive computing session within a CDN 100, such as an onlinepresentation, eLearning lecture or interactive learning session,interactive gaming session, or collaborative remote work session. In anyof these examples, the expression input device 610 may correspond to auser device 106 or 110 of a participant, presenter, collaborator, etc.The expression converter 620 may determine in step 703 whether any ofthe other user devices participating in the online presentation session,eLearning lecture or interactive learning session, interactive gamingsession, or collaborative remote work session, are refreshable brailledisplay devices 630. If a system 600 includes one or more currentlyactive refreshable braille display devices 630 (703:Yes), then theexpression converter 620 may determine that braille output should begenerated and output to these devices 630. For example, if a set of userdevices 106 receiving a live presentation, participating in a real-timecollaborative work session, or receiving other CDN content resources,includes one or more of the braille display devices 630 a-630 c(703:Yes), then the expression converter 620 may convert themathematical expression to braille output as described below in step704. However, if no braille readers are using refreshable brailledisplay devices 630 to receive and participate in the communicationsession (703:No), then the expression converter 620 may determine not togenerate braille output. For example, if the set of user devices 106receiving a live presentation, participating in a real-timecollaborative work session, or receiving other CDN content resources,includes only visual display devices 640 a-640 c and no braille displaydevices 630 (703:No), then the expression converter 620 may determinethat there is no need to convert the receive mathematical expressioninput data into braille output in step 704.

In some embodiments, step 703 may be optional, and the expressionconverter 620 may determine that braille output should be generated inall cases. Thus, in such embodiments, the expression converter 620 neednot be aware of which user devices 106 (if any) are receiving,participating in the session, or otherwise consuming the contentresources including the mathematical expression input received in step701, but instead may simply perform all conversion processes describedherein in response to each received mathematical expression input.

In step 704, the expression converter 620 may execute one or moreconversion processes on the content markup data generated in step 702 toconvert the content markup to braille output characters. To perform theconversion in step 704, the expression converter 620 may invoke thecontent markup-to-braille converter 622 and/or the applicable expressionconversion rules for content markup-to-braille conversion from anexpression conversion rules data store 625. The output generated by thecontent markup-to-braille converter 622 in step 704 may correspond toNemeth Braille Code for Mathematics, and/or any other braillemathematical notation. Converter 622 may be configured to parse contentmarkup generated in step 702, and identify the proper braille charactersand the appropriate format and presentation order for the braillecharacters to be displayed. The conversion rules integrated withinand/or applied by the content markup-to-braille converter 622 mayinclude one-to-one character mapping rules (i.e., rules mapping a singlemathematical symbol to a single braille symbol), one-to-many charactermapping rules (i.e., rules mapping a single mathematical symbol tomultiple braille symbols), and/or various special braille encoding rulesfor mathematical expressions that do not correspond to one-to-one orone-to-many character mapping rules. Such special braille encodingrules, discussed below in more detail in reference to FIGS. 12-14, maybe used to provide uniformity of presentation of braille output and toreduce or eliminate ambiguity for the represented mathematicalexpression.

In step 705, the expression converter 620 may transmit the brailleoutput characters (also referred to as braille symbols) to theappropriate refreshable braille display devices 630. As noted above, therecipient devices 630 in step 705 may correspond to the set of brailledisplay devices 630 participating in an online presentation session,eLearning lecture or interactive learning session, interactive gamingsession, or collaborative remote work session, etc. Accordingly, thebraille conversion in step 704 and transmission in step 705 may beperformed in real-time or near real-time with respect to themathematical expression input received in step 701, so that refreshablebraille display devices 630 may receive and output the converted braillecharacters in real-time or near real-time with respect to thepresentation, collaborative work session, etc.

In step 706, the expression converter 620 may determine whether or notany visual display devices 640 are present in the system 600 to receivethe mathematical expression output. Thus, step 706 may be performedsimilarly to step 703. For example, the expression converter 620 maydetermine in step 706 whether any user devices 106 participating in anonline presentation session, eLearning lecture or interactive learningsession, interactive gaming session, or collaborative remote worksession, are visual display devices 640. If a system 600 includes one ormore currently active visual display devices 640 (706:Yes), then theexpression converter 620 may determine that visual mathematicalexpression output (e.g., presentation markup) should be generated andoutput to these devices 640. However, if no visual display devices 640are currently receiving and/or participating in the communicationsession (706:No), then the expression converter 620 may determine not togenerate presentation markup as output. Additionally, as described abovefor step 703, step 706 also may be optional in some embodiments, and theexpression converter 620 may determine that visual output (e.g.,presentation markup) should be generated in all cases. Thus, in suchembodiments, the expression converter 620 need not be aware of whichuser devices 106 (if any) are receiving, participating in the session,or otherwise consuming the content resources including the mathematicalexpression input received in step 701.

In step 707, the expression converter 620 may execute one or moreconversion processes on the content markup data generated in step 702 toconvert the content markup to presentation markup. To perform theconversion in step 707, the expression converter 620 may invoke thecontent markup-to-presentation markup converter 623 and/or theapplicable expression conversion rules for contentmarkup-to-presentation markup conversion from an expression conversionrules data store 625. In contrast to the content markup, thepresentation markup for a mathematical expression may define a specifiedformatting, style, and/or layout for representing mathematicalexpressions based on the visual rendering and displaying of theexpressions on a display screen. Thus, presentation markup may bestructured using elements such as rows, characters/symbols within rows,superscript and subscript designations, square root symbols, and othersimilar elements which define the visual rendering of the expression,rather than elements corresponding to the expression tree structure ofthe mathematical expression. Examples of different presentation markupdefinitions that may be used in step 707 include any of presentationmarkup specifications within the Mathematical Markup Language (MathML)editions and/or versions defined by the World Wide Web Consortium (W3C),although it should be understood that these examples are illustrativeonly and non-limiting, and that other presentation markup definitions orspecifications may be used in other examples. For instance, referringbriefly to FIGS. 10B and 11B, two specific examples of presentationmarkup are shown that may be generated in step 707, based on contentmarkup. The presentation markup shown in these examples may correspondto the content markup shown in FIGS. 10A and 11A, respectively, and tothe mathematical expressions shown above each text box in theseexamples.

In step 708, the expression converter 620 may transmit the presentationmarkup output to the appropriate visual display devices 640. As notedabove, the recipient devices 640 in step 708 may correspond to the setof visual display devices 640 participating in an online presentationsession, eLearning lecture or interactive learning session, interactivegaming session, or collaborative remote work session, etc. Accordingly,the conversion to presentation markup in step 707 and transmission instep 708 may be performed in real-time or near real-time with respect tothe mathematical expression input received in step 701, so that visualdisplay devices 640 may receive and output the converted presentationmarkup in real-time or near real-time with respect to the presentation,collaborative work session, etc.

Referring now to FIG. 8, a flow diagram is shown illustrating a processof converting mathematical expression data received from a braille inputdevice to braille output and/or visual display output. In someembodiments, the steps described in connection with FIG. 8 may besimilar or identical to the steps described in FIG. 7. However, incontrast to the FIG. 7, the example described in FIG. 8 specificallydescribes scenarios in which the mathematical expression input receivedincludes braille input characters received from a braille input device430. As with FIG. 7, discussed above, the steps in this process may beperformed by components in the mathematical expression conversionsystems 600 described above, such as an expression converter 620 and/orvarious input and output devices 610, 630, and 640, although the variousfeatures and processes described herein need not be limited to thespecific systems and hardware implementations described above in FIGS.1-6.

In step 801, mathematical expression input data is received, forexample, by an expression converter 620. In this example, the input datacorresponds to one or more braille characters received from a brailleinput device. As discussed above, certain refreshable braille displaydevices 430 may also be braille input devices, and such devices 430 mayinclude braille keyboards and/or other specialized components forreceiving mathematical expression input from visually impaired users. Inother cases, braille input devices need not include refreshable brailledisplays.

In some embodiments, step 801 may be similar or identical to step 701,and any of the mathematical symbols and mathematical expressionsdiscussed above in reference to stop 701 may similarly be received instep 801. For instance, if the expression input device 610 discussedabove is a braille input device 610, then the mathematical expressioninput data received in step 701 may correspond to braille inputcharacters, but all other steps discussed above in FIG. 7 may beperformed similarly or identically. In other cases, the braille inputreceived in step 801 may correspond to a response or editing of theinitial mathematical expression input by a different user in step 701.For example, the steps described in FIG. 7 may correspond to a processin which a first user (e.g., a presenter, teacher, trainer, co-worker,etc.) inputs a mathematical expression into a first device 610 within animplementation of an educational or professional training softwaresystem 600, an interactive gaming system 600, an online presentationsystem 600, a collaborative work environment 600, or the like, and themathematical expression input is then converted and transmitted to oneor more additional display devices 630 and/or 640. Continuing thisexample, after the steps described in FIG. 7 are performed, the stepsdescribed in FIG. 8 may be performed, corresponding to a process inwhich a different user responds to the initial communication of thefirst user by altering or editing the initial mathematical expression(e.g., correcting the work of the first user, making a suggestion of adifferent mathematical expression, etc.), or by providing a differentresponsive mathematical expression (e.g., answering a math questioncontained in the initial communication), using a braille input device(e.g., 430).

In step 802, the expression converter 620 may execute one or moreconversion processes on the braille input characters received in step801 to convert the mathematical expression to content markup. To performthe conversion in step 802, the expression converter 620 may invoke thebraille-to-content markup converter 624 and/or the applicable expressionconversion rules for braille-to-content markup conversion from anexpression conversion rules data store 625. In some embodiments, thebraille-to-content markup conversion may be similar to themath-to-content markup conversion described above in step 701, althoughbraille characters may be used as the initial input to the conversionprocess rather than ASCII or Unicode characters. Additionally, in someembodiments, the braille-to-content markup conversion process may besimilar but converse to the content markup-to-braille conversiondescribed above in 704. However, as noted above, each of the separateconverters 621-624 and/or conversion processes described herein may useseparate and unique conversion rules, including character mapping rules(e.g., one-to-one, one-to-many, and many-to-one character mappings) aswell as special encoding and decoding rules for mathematical expressionsthat do not correspond to character mappings.

In step 803, the expression converter 620 may determine whether or notany visual display devices 640 should receive the mathematicalexpression output. Thus, step 803 may be similar or identical to step706, discussed above. For example, the expression converter 620 maydetermine in step 803 whether any user devices 106 participating in anonline presentation session, eLearning lecture or interactive learningsession, interactive gaming session, or collaborative remote worksession, are visual display devices 640. If a system 600 includes one ormore currently active visual display devices 640 (803:Yes), then theexpression converter 620 may determine that visual mathematicalexpression output (e.g., presentation markup) should be generated andoutput to these devices 640. In such cases, in step 804, the expressionconverter 620 may execute one or more conversion processes on thecontent markup data generated in step 802 to convert the content markupto presentation markup. Then, in step 805, the expression converter 620may transmit the presentation markup output to the appropriate visualdisplay devices 640. Steps 804 and 805 may be similar or identical tosteps 707 and 708, discussed above.

In step 806, the expression converter 620 may execute one or moreconversion processes on the content markup data generated in step 802 toconvert the content markup to braille output characters. Then, in step807, the expression converter 620 may transmit the braille outputcharacters to the appropriate refreshable braille display devices 630.Thus, steps 806 and 807 may be similar or identical to steps 704 and705, discussed above.

It should be noted that the conversions described in this exampleinclude a first conversion from braille input characters to contentmarkup in step 802, and a second conversion from the content markup backto braille output characters in step 806. Performing both of theseconversions may be advantageous in some cases, for example, if themathematical expression input received from the user device in step 801is potentially mathematically ambiguous, or if it is expressed in anon-standard and/or non-uniform way. For instance, if the user inputs amathematical expression such as “a+b” or “a(b)” it may be unclearwhether the expression represents a number or a function. Similarly, auser may input any number of potentially ambiguous mathematicalexpressions, such as “sin xy”, “sin̂2 x”, “a/2n”, or various otherambiguous expressions that have multiple possible intended meanings Byconverting the braille input characters first to content markup in step802, and then back to braille output characters in step 806, anypotential ambiguity within the expression may be resolved during theconversion to content markup in step 802, and the braille outputrepresentation of the mathematical expression transmitted in step 807will thus be identical for all braille display devices 430 a-430 c. Byresolving the potential ambiguities in the mathematical expressionduring the conversion processes, and by providing for uniformity inbraille output display, situations may result in which the braille inputcharacters received from a braille device 430 a in step 801 do notprecisely match the braille output characters transmitted back to thesame braille device 430 a in step 807.

Moreover, it may be assumed in this example that the braille inputdevice from which the mathematical expression input was received in step801 will also be a refreshable braille display device on which themathematical expression should be displayed in step 807. Accordingly,this example process does not include a step corresponding to step 703,in which the expression converter 620 determines whether there are anyrefreshable braille display devices 630 that should receive the brailleoutput characters.

Referring now to FIG. 9, a flow diagram is shown illustrating a processof converting mathematical expression data received from a visualdisplay device to visual display output and/or braille output. The stepsdescribed in connection with FIG. 8 may be similar or identical to thesteps described in FIGS. 7-8. In contrast to the FIG. 8, the exampledescribed in FIG. 9 specifically describes scenarios in which themathematical expression input received includes non-braille inputcharacters received from the input components of a visual display device440, such as a keyboard, mouse, touchscreen, and/or specializedmathematics software. As with FIGS. 7-8, discussed above, the steps inthis process may be performed by components in the mathematicalexpression conversion systems 600 described above, such as an expressionconverter 620 and/or various input and output devices 610, 630, and 640,although the various features and processes described herein need not belimited to the specific systems and hardware implementations describedabove in FIGS. 1-6.

In step 901, mathematical expression input data is received, forexample, by an expression converter 620. In this example, the input datacorresponds to one or more non-braille mathematical symbols (orcharacters) received from the inputs of a visual display device 640.Step 901 may be similar or identical to step 701, and any of themathematical symbols and mathematical expressions discussed above inreference to stop 701 may similarly be received in step 901. In somecases, the steps described in FIG. 7 may correspond to a process inwhich a first user (e.g., a presenter, teacher, trainer, co-worker,etc.) inputs a mathematical expression into a first device 610 within animplementation of an educational or professional training softwaresystem 600, an interactive gaming system 600, an online presentationsystem 600, a collaborative work environment 600, or the like, and themathematical expression input is then converted and transmitted to oneor more additional display devices 630 and/or 640. Continuing thisexample, after the steps described in FIG. 7 are performed, the stepsdescribed in FIG. 9 may be performed, corresponding to a process inwhich a different user responds to the initial communication of thefirst user by altering or editing the initial mathematical expression(e.g., correcting the work of the first user, making a suggestion of adifferent mathematical expression, etc.), or by providing a differentresponsive mathematical expression (e.g., answering a math questioncontained in the initial communication), using a visual display device440.

In step 902, the expression converter 620 may execute one or moreconversion processes on the mathematical input characters received instep 901 to convert the mathematical expression to content markup. Theconversion processes used in step 902 may be similar or identical to theconversion processes used in step 702, discussed above, the expressionconverter 620 may invoke the same math-to-content markup converter 621and/or the same applicable expression conversion rules formath-to-content markup conversion from an expression conversion rulesdata store 625.

In step 903, the expression converter 620 may determine whether or notany refreshable braille display devices 630 should receive themathematical expression output. Thus, step 903 may be similar oridentical to step 703, discussed above. For example, the expressionconverter 620 may determine in step 903 whether any user devices 106participating in an online presentation session, eLearning lecture orinteractive learning session, interactive gaming session, orcollaborative remote work session, are braille display devices 630. If asystem 600 includes one or more currently active braille display devices630 (903:Yes), then the expression converter 620 may determine that abraille representation of the mathematical expression should begenerated and output to these devices 630. In such cases, in step 904,the expression converter 620 may execute one or more conversionprocesses on the content markup data generated in step 902 to convertthe content markup to braille output characters. Then, in step 905, theexpression converter 620 may transmit the braille output charactersrepresenting the mathematical expression to the appropriate refreshablebraille display devices 630. Steps 904 and 905 may be similar oridentical to steps 704 and 705, discussed above.

In step 906, the expression converter 620 may execute one or moreconversion processes on the content markup data generated in step 802 toconvert the content markup to presentation markup. Then, in step 907,the expression converter 620 may transmit the presentation markup outputto the appropriate visual display devices 640. Thus, steps 906 and 907may be similar or identical to steps 707 and 708, discussed above.

As discussed above for FIG. 8, it should be noted that the conversionsdescribed in FIG. 9 include a first conversion from mathematical inputcharacters to content markup in step 902, and a second conversion fromthe content markup back to visual output (e.g., presentation markup) instep 906. Performing both of these conversions may be advantageous forresolving potential ambiguities within the mathematical expression inputdata received in step 901, as well providing a standard and uniform setof visual output to all visual display devices 640, as discussed above.Moreover, it may be assumed in this example that the visual input devicefrom which the mathematical expression input was received in step 901will also be a visual display device on which the mathematicalexpression should be displayed in step 907. Accordingly, this exampleprocess does not include a step corresponding to step 706, in which theexpression converter 620 determines whether there are any visual displaydevices 440 that should receive the presentation markup representing themathematical expression.

Referring now to FIGS. 12, 13, and 14A-14C, several example userinterface screens are shown from a braille conversion application. Thebraille conversion application shown in these examples may beimplemented using an underlying expression converter 620, discussedabove, which may be configured to convert braille mathematicalexpression input to text (e.g., presentation markup) output, as well astext or other visual mathematical expression input to braille output. Inthese examples, each user interface displays an upper text input/outputwindow 1210 and a lower braille input/output window 1220. When a userinputs mathematical characters into the text window 1210 using the mathsymbol selection panel 1230, the underlying expression converter 620 mayperform one or more of the mathematical expression conversion processesdescribed below in reference to FIGS. 7-9. For example, in response toone or more mathematical characters being input, revised, or deletedfrom the text window 1210, the expression converter 620 may perform afirst conversion of the entire expression within the text window 1210from mathematical input into content markup, followed by a secondconversion of the newly generated content markup into braille outputcharacters for display within the braille window 1220. Similarly, inresponse to one or more braille characters being input, revised, ordeleted from the braille window 1220, the expression converter 620 mayperform a first conversion of the entire mathematical expression withinthe braille window 1220 from braille input characters into contentmarkup, followed by a second conversion of the newly generated contentmarkup into presentation markup for display within the text window 1210.Additionally, it should be understood that the braille input/outputwindow 1220 is displayed visually in these examples in order to betterillustrate the conversion processes and the braille encoding anddecoding rules implemented by the expression converter 620. However, inother examples, the braille window 1220 may correspond to one or moreseparate refreshable braille display devices 430, and the text window1210 may correspond to one or more visual display devices 440.

In FIG. 12, the four mathematical characters “a(x)” have been inputeither into the text window 1210 or the braille window 1220, and theexpression converter 620 has performed the conversion processesdescribed above to generate and transmit the corresponding output intothe other window. FIG. 13 shows a more complex mathematical expression,but a similar or the same conversion processes may be used by theexpression converter 620 to convert from braille input to visual output,or vice versa. In these examples, symmetrical character mapping rulesmay be applied by the expression converter 620 during thebraille-to-content markup and content markup-to-braille conversions. Forexample, certain mathematical symbols may be mapped to single braillecharacters by the expression converter 620, while other mathematicalsymbols may be mapped to combinations of braille characters.

FIGS. 14A-14C show another example in which a user is inputting asequence of mathematical characters either into the text window 1210 orthe braille window 1220, and the expression converter 620 has beeninvoked at each step to perform the conversion processes described aboveto generate and transmit the corresponding output into the other window.These figures also illustrate examples of special braille encoding anddecoding rules that do not correspond to character mapping rules.Special rules may be designed and applied by the expression converter620 to avoid ambiguities within the converted mathematical expressions,and to address special braille encoding and decoding situations that mayarise. As noted above, each different converter 621-624 and/orconversion process described herein may invoke its own set of specialrules for braille encoding and decoding.

In FIG. 14A, an absolute value operation “|x|” is shown in text window1210, and the three corresponding braille characters (i.e., a firstvertical bar braille character (dot-1256), a braille letter “x”, andthen a second vertical bar braille character (dot-1256)) are shown inthe braille window 1220. In FIG. 14B, two consecutive absolute valueoperations “|x∥y|” are shown in text window 1210, and the correspondingbraille window 1220 shows the corresponding braille characters for thetwo consecutive absolute value operations, separated by a multi-purposeindicator 1221. As seen in this example, the multi-purpose indicator1221 is represented by a single dot braille character (dot-5). Thereason for the multi-purpose indicator 1221 is to resolve avoid thepotential ambiguity of two consecutive vertical bar braille characters(e.g., (dot-1256) (dot-1256)), which may otherwise indicate amathematical norm operator “∥”. It should be understood that thisexample is illustrative only and non-limiting, and that othermulti-purpose indicators 1221 may be inserted during the processes ofconverting mathematical expressions into braille or from braille, inorder to avoid any other potential ambiguities within the braille outputor visual output. Additionally, although the multi-purpose indicator1221 is a single dot braille character (dot-5), it should be understoodthat other characters or combinations of characters may be used to avoidambiguities in other examples. In FIG. 14C, additional input has beenadded to transform the expression in FIG. 14B to an equation“|x∥y|=∥□∥”. This figure again shows the use of the multi-purposeindicator 1221 to distinguish two consecutive absolute value operatorsfrom a norm operator. Additionally, in this example, a special spacingrule has been applied by the expression converter 620, in which a spacecharacters 1212 and 1222 have been added to each side of the equal signin both the visual output and the braille output. The spacing rule inthis example may be enforced by the expression converter 620 to avoidambiguity, increase readability of the braille output characters, andalso to provide uniformity between the braille output and the visualoutput.

Any number of other examples of special braille encoding and/or decodingrules may be implemented in other embodiments. For example, specialrules for converting mathematical expressions containing nestedfractions into braille and/or from braille may be implemented in somecases. For instance, in some embodiments, an expression converter 620may convert a simple fraction (i.e., a fraction that does not itselfcontain a fraction) using a fraction indicator braille character, afraction bar braille character, and a fraction terminator braillecharacter, while a complex fraction (i.e., a fraction that contains atleast one other fraction) may use additional nested fraction indicatorbraille characters. Thus, the simple fraction ¾ may be encoded intobraille as: [fraction indicator character] 3 [fraction bar character] 4[fraction terminator character], while the complex fraction (⅔)/4 may beencoded into braille as [nested fraction indicator character] [fractionindicator character] 2 [fraction bar character] 3 [fraction terminatorcharacter] [nested fraction bar character] 4 [nest fraction terminatorcharacter]. For illustrative purposes only, the fraction indicator maybe represented by the braille character (dot-1456), and the nestedfraction indicator may be represented by the braille character string(dot-6) (dot-1456). To represent each additional enclosed fraction, anadditional (dot-6) braille character may be added to the fractionindicator, the fraction bar, and the fraction terminator, in theseexamples.

A set of special braille encoding and/or decoding rules may beimplemented for handling nested roots, similar to rules discussed abovefor handling nested fractions. For example, additional nested rootindicator braille characters may be defined, and an extra braillecharacter (e.g., dot-46) may be added to the open and close rootindicators for each enclosed (not enclosing) root. Accordingly, thedeeper into a nested root expression, the more dots that will be used toencode the root operator. Additionally, special encoding and/or decodingrules may be used for representing braille subscripts and/orsuperscripts in some embodiments. For example, the standard brailleencodings for superscripts and subscripts may be “x ̂i” and “x sub i”respectively. However, in some cases, abbreviated braille encodings maybe used for subscripts and/or superscripts when the subscript orsuperscript is a numeral. Thus, expressions like “x ̂2” and “x sub 4”might each be represented by only two braille characters in some cases.

It should be understood that the above examples of special brailleencoding rules for mathematical expressions are illustrative only andnon-limiting, and that other special braille encoding rules may beimplemented in other examples in order to avoid ambiguities within themathematical expressions, improve readability of the braille output orvisual output, or provide uniformity between the braille and visualoutputs. Additionally, different sets of encoding and decoding rules maybe implemented for conversions between braille input and visual output,and vice versa.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

1. A braille generator and converter system comprising: one or morerefreshable braille display devices, each refreshable braille displaydevice comprising: a plurality of electromechanical braille cells, eachelectromechanical braille cell comprising a plurality of output dotsconfigured to raise and lower in response to voltages applied at theelectromechanical braille cell; a processing unit comprising one or moreprocessors; and memory coupled with and readable by the processing unitand storing therein a set of instructions which, when executed by theprocessing unit, causes the refreshable braille display device to:receive data corresponding to one or more braille output characters;determine an electromechanical braille cell for displaying each of thebraille output characters; and transmit control instructions to each ofthe determined electromechanical braille cells to display the one ormore braille output characters; an expression input device, comprising:an I/O subsystem configured to receive mathematical expression inputdata via one or more input devices integrated into or connected with theexpression input device; a processing unit comprising one or moreprocessors; and memory coupled with and readable by the processing unitand storing therein a set of instructions which, when executed by theprocessing unit, causes the expression input device to: receive inputdata via the I/O subsystem corresponding to a mathematical expression;and transmit data representing the mathematical expression to anexpression converter; and the expression converter, the expressionconverter comprising: a processing unit comprising one or moreprocessors; and memory coupled with and readable by the processing unitand storing therein a set of instructions which, when executed by theprocessing unit, causes the expression converter to: receive the datafrom the expression input device representing the mathematicalexpression during an interactive computing session; retrieve a first setof data conversion rules for converting mathematical expressions tocontent markup; convert the data representing the mathematicalexpression to a content markup representation of the mathematicalexpression, using the first set of data conversion rules; determinewhether one or more of the refreshable braille display devices areconfigured to receive braille output within the interactive computingsession; in response to determining that at least one of the refreshablebraille display device is configured to receive braille output withinthe interaction computing session: retrieve a second set of dataconversion rules for converting content markup to braille output,wherein the second set of data conversion rules includes at least onebraille encoding rule that does not correspond to a character mappingrule; convert the content markup representation of the mathematicalexpression to a set of braille output characters, using the second setof data conversion rules, wherein the set of braille output charactersincludes at least one braille character that does not correspond to anycharacter in the received input data or in the content markuprepresentation of the mathematical expression; and transmit the set ofbraille output characters to the at least one refreshable brailledisplay devices configured to receive braille output within theinteraction computing session.
 2. (canceled)
 3. The braille generatorand converter system of claim 1, the memory of the expression converterstoring therein further instructions which, when executed by theprocessing unit, causes the expression converter to: receive a set ofbraille input characters from a first refreshable braille displaydevice, the set of braille input characters representing an updatedmathematical expression; retrieve a third set of data conversion rulesfor converting braille input to content markup; and convert the receivedbraille input characters representing the updated mathematicalexpression to a content markup representation of the updatedmathematical expression, using the third set of data conversion rules.4. The braille generator and converter system of claim 3, furthercomprising: one or more visual presentation devices, each visualpresentation device comprising: a visual display screen configured torender representations of mathematical expressions; a processing unitcomprising one or more processors; and memory coupled with and readableby the processing unit and storing therein a set of instructions which,when executed by the processing unit, causes the visual presentationdevice to: receive data corresponding to presentation markuprepresentations of mathematical expressions; parse the receivedpresentation markup representations of mathematical expressions; andrender mathematical expressions corresponding to the receivedpresentation markup representations on the visual display screen,wherein the memory of the expression converter stores therein furtherinstructions which, when executed by the processing unit, causes theexpression converter to: identify one or more of the visual presentationdevices to receive presentation markup output corresponding to theupdated mathematical expression; retrieve a fourth set of dataconversion rules for converting content markup to presentation markup;convert the content markup representation of the updated mathematicalexpression to a presentation markup representation of the updatedmathematical expression, using the fourth set of data conversion rules;and transmit the presentation markup representation of the updatedmathematical expression to the identified one or more visualpresentation devices.
 5. The braille generator and converter system ofclaim 4, wherein receiving the input data by the expression input devicecomprises receiving input of a first set of characters via the I/Osubsystem of the expression input device, and wherein the presentationmarkup representation of the mathematical expression converted by theexpression converter corresponds to a rendering of the mathematicalexpression including a second set of characters different from the firstset of characters.
 6. The braille generator and converter system ofclaim 1, wherein the memory of the expression converter is configured tostore a plurality of expression converter components, including: amathematical expression input to content markup converter; a braillecharacter input to content markup converter; a content markup to brailleoutput character converter; and a content markup to presentation markupconverter. 7-8. (canceled)
 9. A method, comprising: receiving, by anexpression converter, input data representing a mathematical expressionduring an interactive computing session; retrieving, by the expressionconverter, a first set of data conversion rules for convertingmathematical expressions to content markup; converting, by theexpression converter, the data representing the mathematical expressionto a content markup representation of the mathematical expression, usingthe first set of data conversion rules; determining, by the expressionconverter, whether one or more refreshable braille display devices areconfigured to receive braille output within the interactive computingsession; in response to determining that at least one refreshablebraille display device is configured to receive braille output withinthe interaction computing session: retrieving, by the expressionconverter, a second set of data conversion rules for converting contentmarkup to braille output, wherein the second set of data conversionrules includes at least one braille encoding rule that does notcorrespond to a character mapping rule; converting, by the expressionconverter, the content markup representation of the mathematicalexpression to a set of braille output characters, using the second setof data conversion rules, wherein the set of braille output charactersincludes at least one braille character that does not correspond to anycharacter in the received input data or in the content markuprepresentation of the mathematical expression; and transmitting, by theexpression converter, the set of braille output characters to the atleast one refreshable braille display device, wherein the content markuprepresentation of the mathematical expression is not converted tobraille output characters when there is not at least one refreshablebraille display device configured to receive braille output within theinteraction computing session.
 10. The method of claim 9, furthercomprising: receiving additional input data comprising asingle-character revision to the mathematical expression; determining anupdated mathematical expression based on the single-character revisionto the mathematical expression; and in response to receiving thesingle-character revision to the mathematical expression, and inresponse to the determination that at least one refreshable brailledisplay device is configured to receive braille output within theinteraction computing session: converting the data representing theupdated mathematical expression to a content markup representation ofthe updated mathematical expression, using the first set of dataconversion rules; converting the content markup representation of theupdated mathematical expression to an updated set of braille outputcharacters, using the second set of data conversion rules; andtransmitting the updated set of braille output characters to the atleast one refreshable braille display device. 11-13. (canceled)
 14. Themethod of claim 9, wherein the second set of data conversion rules forconverting content markup to braille output includes a plurality ofbraille encoding rules that do not correspond to character mappingrules, the plurality of braille encoding rules including at least oneof: a braille encoding rule for handling nested fractions; a brailleencoding rule calling for a multipurpose indicator braille symbol; abraille encoding rule for handling subscripts; or a braille encodingrule for handling superscripts.
 15. The method of claim 9, wherein thesecond set of data conversion rules for converting content markup tobraille output includes a braille encoding rule for handling nestedfractions that does not correspond to a character mapping rule, andwherein the set of braille output characters includes a first braillesymbol corresponding to a fraction indicator and a second braille symbolcorresponding to a nested fraction indicator.
 16. A computer-programproduct tangibly embodied in a non-transitory machine-readable storagemedium, including instructions configured to cause one or more dataprocessors to perform actions including: receiving input datarepresenting a mathematical expression during an interactive computingsession; retrieving a first set of data conversion rules for convertingmathematical expressions to content markup; converting the datarepresenting the mathematical expression to a content markuprepresentation of the mathematical expression, using the first set ofdata conversion rules; determining whether one or more refreshablebraille display devices are configured to receive braille output withinthe interactive computing session; in response to determining that atleast one refreshable braille display device is configured to receivebraille output within the interaction computing session: retrieving asecond set of data conversion rules for converting content markup tobraille output, wherein the second set of data conversion rules includesat least one braille encoding rule that does not correspond to acharacter mapping rule; converting the content markup representation ofthe mathematical expression to a set of braille output characters, usingthe second set of data conversion rules, wherein the set of brailleoutput characters includes at least one braille character that does notcorrespond to any character in the received input data or in the contentmarkup representation of the mathematical expression; and transmittingthe set of braille output characters to the at least one refreshablebraille display device, wherein the content markup representation of themathematical expression is not converted to braille output characterswhen there is not at least one refreshable braille display deviceconfigured to receive braille output within the interaction computingsession.
 17. The computer-program product of claim 16, including furtherinstructions configured to cause the one or more data processors toperform actions including: receiving additional input data comprising asingle-character revision to the mathematical expression; determining anupdated mathematical expression based on the single-character revisionto the mathematical expression; and in response to receiving thesingle-character revision to the mathematical expression and in responseto the determination that at least one refreshable braille displaydevice is configured to receive braille output within the interactioncomputing session: converting the data representing the updatedmathematical expression to a content markup representation of theupdated mathematical expression, using the first set of data conversionrules; converting the content markup representation of the updatedmathematical expression to an updated set of braille output characters,using the second set of data conversion rules; and transmitting theupdated set of braille output characters to the at least one refreshablebraille display device.
 18. The computer-program product of claim 16,including further instructions configured to cause the one or more dataprocessors to perform actions including: receiving a set of brailleinput characters from a first refreshable braille display device, theset of braille input characters representing an updated mathematicalexpression; retrieving a third set of data conversion rules forconverting braille input to content markup; and converting the receivedbraille input characters representing the updated mathematicalexpression to a content markup representation of the updatedmathematical expression, using the third set of data conversion rules.19. The computer-program product of claim 18, including furtherinstructions configured to cause the one or more data processors toperform actions including: identifying one or more visual presentationdevices to receive presentation markup output corresponding to theupdated mathematical expression; retrieving a fourth set of dataconversion rules for converting content markup to presentation markup;converting the content markup representation of the updated mathematicalexpression to a presentation markup representation of the updatedmathematical expression, using the fourth set of data conversion rules;and transmitting the presentation markup representation of the updatedmathematical expression to the identified one or more visualpresentation devices.
 20. The computer-program product of claim 19,wherein input data comprises a first set of characters, and wherein thepresentation markup representation of the mathematical expressioncorresponds to a rendering of the mathematical expression including asecond set of characters different from the first set of characters. 21.The method of claim 9, wherein converting the content markuprepresentation of the mathematical expression to the set of brailleoutput characters using the second set of data conversion rulescomprises: applying character mapping rules to the content markuprepresentation of the mathematical expression, to generate acorresponding string of braille characters; analyzing the generatedstring of braille characters to detect one or more potentialambiguities; and using the at least one braille encoding rule that doesnot correspond to a character mapping rule, to resolve the potentialambiguities in the generated string of braille characters.
 22. Themethod of claim 21, wherein detecting the one or more potentialambiguities, and using the at least one braille encoding rule that doesnot correspond to a character mapping rule to resolve the potentialambiguities comprises: determining that two consecutive characters inthe generated string of braille characters correspond to a singlemathematical symbol; and inserting a designated indicator braillecharacter in between the two consecutive characters, in response todetermining that the two consecutive characters correspond to a singlemathematical symbol.
 23. The method of claim 9, wherein the set ofbraille output characters includes a fraction indicator braillecharacter that does not correspond to any character in the receivedinput data, a fraction terminator braille character that does notcorrespond to any character in the received input data, and fraction barbraille character.
 24. The method of claim 23, wherein the set ofbraille output characters further includes a nested fraction indicatorbraille character that is separate from the fraction indicator braillecharacter, and a nested fraction terminator braille character that isseparate from the fraction terminator braille character.
 25. The methodof claim 9, wherein the set of braille output characters includes a rootindicator braille character, and a nested root indicator braillecharacter that is separate from the root indicator braille character.26. The method of claim 9, wherein converting the content markuprepresentation of the mathematical expression to the set of brailleoutput characters using the second set of data conversion rulescomprises: identifying a first character corresponding to a subscript orsuperscript character in the content markup representation of themathematical expression, and a second character in the content markuprepresentation of the mathematical expression to which the firstsubscript or superscript character applies; determining that the firstsubscript or superscript character is an integer; and based on thedetermination that the first subscript or superscript character is aninteger, applying an abbreviated braille encoding rule by which thefirst character, the second character, and the relationship between thefirst character and the second character is represented by only twobraille characters.